Fix cache invalidation for module binaries#6914
Conversation
✅ Deploy Preview for nextflow-docs-staging canceled.
|
|
From the issue referenced #6128, someone wrote a minimal test pipeline: jashapiro/nextflow-resume-test. We can test this fix using the following procedure: Run once:
where Change the module binary and rerun with resume: Expected result with the fix: Before the fix, both processes could be restored from cache even after changing modules/test/resources/usr/bin/hello_test.sh. The included groovy test in the PR should emulate the same behaviour. I have tested the fix using a local build of Nextflow with these changes and that worked. @pditommaso Do you prefer the fix implementation via a different mechanism? If so, can you elaborate how? |
|
If am i not wrong the test does not use wave https://github.com/jashapiro/nextflow-resume-test |
|
Ok, managed to replicate:
|
|
@pditommaso Thanks for replicating! Anything else missing to get this merged? |
|
Yes, this PR requires merging this first #6927 |
|
hey, are we planning to merge this at any time? |
|
Since we decided not to merge #6927 , I think we should go ahead and fix this issue. However I need to review this issue again and make sure this fix is the right fix |
When a module uses local binaries via its `resources/` directory (the `nextflow.enable.moduleBinaries` feature flag), those files were not part of the task hash used for `-resume` caching. As a result, editing a module's `resources` script did not invalidate the cache and `-resume` returned a stale cached result instead of re-executing the task. Add the module resources bundle fingerprint to the task hash key list in `TaskHasher.compute()`, guarded by `session.enableModuleBinaries()` (matching the existing guard in `TaskProcessor.getBinDirs()`), so edits to module binaries correctly force task re-execution while unchanged resources still hit the cache. Note: this does not affect Wave setups, where the resources are baked into the container and already covered by the container fingerprint. Fixes nextflow-io#6128 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Ben Sherman <bentshermann@gmail.com>
0914b92 to
9e2e7eb
Compare
✅ Deploy Preview for nextflow-docs canceled.
|
bentsherman
changed the title
bentsherman
left a comment
bentsherman
left a comment
There was a problem hiding this comment.
Reviewed the original issue and updated the PR with my preferred implementation. It just adds the module resources to the task hash
This PR fixes the specific case of using module binaries without Wave on a shared filesystem
4 participants
Summary
Fix task cache invalidation when using module binaries with Wave.
Task hashing previously considered project-level
bin/scripts but not module-level binaries underresources/usr/bin, which could cause stale cached tasks to be reused after a module binary changed.This change makes task hashing resolve referenced executables from the task bin paths, including module binary directories, so module binary updates invalidate the cache as expected.
Closes #6128
Testing
TaskHasherTest./gradlew :nextflow:test --tests 'nextflow.processor.TaskHasherTest'